The existing fall detection works mainly focus on indoor scenes, and most of them only model people’s body posture features, ignoring background information of the scene and the interaction information between people and the ground. Aiming at the problem, from the perspective of practical application of elevator scene, a fall detection algorithm based on scene prior and attention guidance was proposed. Firstly, elevator historical data was used to automatically learn the scene prior information from people’s trajectories by Gaussian probability distribution modelling. Then, the scene information was taken as a spatial attention mask and fused with the global features of the neural network to focus on local information of the ground area. After that, the fused local and global features were further aggregated using adaptive weighting method to improve the robustness and discriminative ability of the generated features. Finally, the features were fed into a classifier module consisting of a global average pooling layer and a fully connected layer to perform the fall prediction and classification. Experimental results show that the detection accuracy of the proposed algorithm on the self-built elevator scene dataset Elevator Fall Detection Dataset and the public UR Fall Detection Dataset reached 95.36% and 99.01% respectively, which is increased by 3.52 percentage points and 0.61 percentage points respectively compared with that of ResNet50 with complicated network structure. It can be seen that proposed attention mechanism with Gaussian scene prior guidance can make the network focus on information of the ground area, which is more conducive to detect fall events. By using it, the detection model has high accuracy, and the algorithm meets the real-time application requirements.